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ABSTRACT: The diffusion of Building Information Modeling (BIM) and advanced visualization technologies in 
the increasingly digitalised construction sector is fostering the development and implementation of disruptive 
approaches for workforce Health and Safety (H&S) training. Project-specific risks, safety procedures and 
information can be administered through immersive Virtual Reality (VR) experiences where construction site 
environments and activities are reproduced without exposing the trainees to real hazards. However, despite 
numerous research and industry applications demonstrating the potential benefits of these technologies, a 
standardized framework and methodology for the evaluation of VR safety training effectiveness for construction 
workers is still lacking hence hindering its large scale-adoption and recognition from policymakers. Within the 
scope of previous authors contributions on the development and implementation of BIM-based VR experiences for 
construction workers’ safety training, this paper aims to address the evaluation of their effectiveness proposing a 
novel semi-qualitative approach based on the integration of trainees’ subjective and objective data. A post- 
experience evaluation questionnaire is developed to collect trainees’ direct and qualitative feedback about the 
experience immersivity and perceived safety content transfer. Furthermore, the integration with trainees’ spatial 
tracking data is proposed to complement the qualitative feedback with the quantitative evaluation of their use of 
the virtual space for safety training purposes. The application of the presented approach in case study is currently 
undergoing and the related results will be subject of future contributions. 


Keywords: Virtual Reality (VR), Construction worker, Safety training, Evaluation, Spatial tracking, Heatmap 
visualization, Survey 


1. INTRODUCTION 


Despite recent technological innovations and policy improvements in workforce Health and Safety (HS) are 
contributing to a low but steady reduction in accident rates, the construction industry is still one of the most 
dangerous, accounting for one fifth of yearly workplace fatal accidents in the European Union alone (Eurostat, 
2022). In this regard, the growing adoption of real-scale immersive visualizations of complex site scenarios and 
construction activities enabled by Building Information Modelling (BIM) and Virtual Reality (VR) technologies 
are supporting HS managers in the early identification and mitigation of safety risks (Babalola et al., 2023). 
Moreover, since their early applications, it has been acknowledged that immersive VR simulations of project- 
specific site layouts and activities can improve the transfer of safety contents and preventive procedures to the 
trainees, while empowering their awareness in later real-site hazardous contexts (Rokooei et al., 2023). However, 
the administration of VR experiences for construction workers safety training is far from substituting traditional 
methods (e.g., slides) and is still confined to a minor share of early adopters. In fact, while economic and technical 
barriers have progressively shrunk, the lack of standardized frameworks and methods for the evaluation of the 
effectiveness of construction site VR training still stands as a major obstacle for its recognition from policy makers 
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and hence for their large-scale adoption in the industry. In this context, several contributions have shown how the 
quantification of the trainee’s ability to perform cognitive and practical tasks in the VR environment (e.g., hazard 
identification, activity simulation) can be used to objectively assess the training effectiveness (Li et al., 2018). 
Nonetheless, the qualitative evaluation of VR training, collected via post-experience questionnaires administered 
to the trainees, is often overlooked or tailored on a specific application, so that its reuse in other case studies is 
impractical. 


To address the mentioned open issues in the evaluation of VR safety training experiences for construction workers, 
this paper proposes a novel approach based on the integration of trainee’s subjective perception and objective VR 
spatial usage. The former is collected through a questionnaire administered to the trainee after the experience and 
divided in five inquiry areas. For the latter, the acquisition of the users’ spatial track during the VR experience and 
its restitution in a BIM environment through an heatmap visualization is proposed to evaluate the trainees’ use and 
understanding of the virtual environment in relation to safety training purposes. The present work stems from 
previous authors’ contributions in the development and administration of BIM-based immersive VR site 
simulations to construction workers for HS training and management purposes and is currently being tested in an 
infrastructure project case study whose results will be subject of future contributions (Getuli, Capone, Bruttini, & 
Sorbi, 2020; Getuli et al., 2018, 2021, 2022). 


2. BACKGROUND 


The use of virtual reality as a safety training technology is gaining attention in the construction industry. While 
many of the current studies mainly focus on the development of VR-based safety training programmes, it is 
noticeable that there is still a lack of research focusing on assessing its effectiveness. In this section, an overview 
of the state of the art with regard to the use of immersive virtual reality used in the realisation of training sessions 
for workers is given and, finally, the identified open problems and obstacles to implementation that this research 
aims to address are reported. 


2.1. BIM and Virtual Reality for construction workers’ safety training 


Increasing use of BIM is favouring the adoption of VR in the Archtiecture, Engineering and Construction (AEC) 
sector. Typical applications for VR include construction safety planning and training (Azhar, 2017), production 
planning and design review sessions (Wolfartsberger, 2019). 


In recent years, several studies have shown that BIM models can be used to represent construction site layouts and 
extract data for space and activity optimisation (Tao et al., 2022) and apply automated safety rule checking to 
simplify hazard recognition and assessment and risk assessment activities (I. Kim et al., 2020). Most of all, the 
spatial understanding and information visualisation capabilities provided by construction site BIM models have 
been harnessed to transfer general and project-specific HS knowledge with the implementation of VR technologies 
for the reproduction of construction site scenarios and activities for worker safety training and site planning (Getuli 
& Capone, 2018). 


Several studies investigated the use of virtual reality to allow construction workers and supervisors to have easy 
access to the BIM through a simple VR interface and also for the education of students, defining an efficient 
educational tool by integrating VR application and BIM model information to develop 3D, 4D, and 5D simulations 
(Esfahani, 2023) and offering suggestions to AEC educators and students in implementing BIM-into-VR in 
different courses. 


The use of virtual reality as a safety training technology is gaining attention in many different fieds: using a fully 
immersive VR has been shown to offer numerous benefits in terms of the effectiveness of health and safety training 
including risk assessment, machinery and/or process operation training in various industries (Toyoda et al., 2022). 


VR is gaining attention also in the construction industry, but while many of the current studies mainly focus on 
the development of VR-based safety training programmes, it is noticeable that there is still a lack of research 
focusing on improving its effectiveness. It has been shown that telepresence experienced through VR and learners' 
perception of the risk of occupational accidents significantly influence their satisfaction with VR-based safety 
training, thus influencing its effectiveness (Yoo et al., 2023). 
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2.2. Construction workers’ safety training evaluation 


The evaluation of the effectiveness of simulations developed in VR during the training sessions or the validation 
of the experience carried out is a key step to identify different problems and to be able to solve certain situations 
by fully improving the training activity for trainees, but while VR-based training has been proven to improve 
learning effectiveness over conventional methods, there is a lack of study on its learning effectiveness due to the 
implementation of training modes. It is known that BIM and VR for safety training of construction workers are 
useful in many contexts (Afzal & Shafiq, 2021), although there is still no standardized method to evaluate the 
effectiveness of safety content transfer. In fact, most of the proposed methods are specific to individual case studies 
or applications in different fields, and the effectiveness of the training administered as perceived by the trainee and 
the spatial understanding of the worksite scenario and activity in VR is often overlooked. 


The evaluation of the user experience in virtual environments can be done either with subjective methods, such us 
questionnaires (H. K. Kim et al., 2018) or with objective methods, like eye-tracking, or brain activity 
measurements (Hertweck et al., 2019). 


Regarding to the evaluation of the VR safety training of construction workers, several survey methods were 
proposed, including the creation of a questionnaire containing open and closed questions to evaluate various 
aspects of the VR interface on a scale of 1 (poor) to 5 (excellent) with subsequent collection of further data through 
observations and conversations with participants during and after the VR tests (Johansson & Roupé, 2019). 
Questionnaires are a widely used and well-known tool to collect user feedback and changes in mental states during 
various activities (Robinson, 2018) including VR applications. 


Although the use of objective measurement methods is promising, questionnaires are the most frequently used tool 
in user experience studies for VR. These questionnaires can be used as pre-, real-time, or post-assessment methods. 
In pre-surveys, the user is not immersed in a virtual environment: this can lead to a less dominant difference 
between VR and the traditional desktop presentation of questionnaires. After immersion into the virtual 
environment, however, it is important to investigate the influence of the type of questionnaire presentation on user 
experience (Safikhani et al., 2021). 


Another important topic is the movement or spatial tracking of workers during VR simulation of construction 
activities, that has been shown to be useful for ergonomic evaluation of workstations or assembly procedures 
(Getuli, Capone, Bruttini, & Isaac, 2020). 


Some studies demonstrate the effectiveness of VR simulations of assembly lines and task scenarios in an 
ergonomic approach to workplace design, aimed to optimize the production and the human-machine interaction 
(Caputo et al., 2018). Through the collection and analysis of the position tracking of a worker (Michalos et al., 
2018) and human motion and posture tracking systems is possible to obtain reliable and repeatable measures to be 
used for evaluation of activity and workplace-related working postures. 


2.3. Open issues 


Although BIM and VR for construction workers’ safety training has been proven beneficial in many studies, there 
is still a lack of a standardized method to evaluate safety contents transfer effectiveness. Most methods are specific 
to single case studies or applications; they are used to assess the training on objective quantification of the trainee 
ability to accomplish tasks in the VR environment but overlook trainee’s perceived effectiveness of the 
administered training and spatial understanding of the construction site scenario and activity in the VR, and ignore 
the subjective worker perception. 


Movement or spatial tracking of the trainees/workers during VR simulation of working activities has been proven 
beneficial for the ergonomic assessment of workstation or assembly procedures but not yet to evaluate the spatial 
understanding of the trainee in VR safety training experience that could instead be leveraged in the safety content 
transfer evaluation of HS training VR experience. 


3. PROPOSED APPROACH 


The aim of this paper is to propose an evaluation method of VR safety training experiences for construction 
workers that is based on the integration of a qualitative survey of the experience and spatial tracking data of 
participants. To this end, first a dedicated questionnaire was developed to evaluate both the immersiveness of VR 
experiences and the effectiveness of the safety content presented. Then a study on the benefits of tracking and 
visualizing trainees' use of space in the virtual environment is proposed. Finally, the proposed approach was 
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SECTION A - EXTENDED REALITY TECHNOLOGIES IN CONSTRUCTION 


validated within a case study previously developed by the authors and used for BIM-based construction worker 
safety training in VR. 


As mentioned above, the proposed approach is based on the integration of two main categories of data, as 
illustrated in Figure 1. The overall goal is to collect a set of data from trainees experiencing immersive VR training, 
aimed at evaluating the effectiveness of the virtual experience. The detailed description of the types of data to be 
collected and the specific purpose for which they need to be obtained then follows in Sections 3.1 - 3.2. 


DATA CATEGORY DATA TYPE PURPOSE 


. Result clustering and indication of 
7 ID & Demographics the worker's data 
Evaluation of trainee's satisfaction 


—> Immersivity a a s a 
and “motion sickness” symptoms 


Evaluation of training virtual 


ere Contents Effectiveness a 2 i 
Qualitative Data representation 
Supervised / Administered : : . 
Questionnaire Training Experience Effectiveness ——> Effectiveness of the immersive VR 


training experience 
Future Developments —— New contents addition 


Collection of trainees’ personal 


-> Suggestions (Open Questions) — > impressions and ideas 


Collect information on the path 


[pg Worker Spatial Track followed by trainees 


Quantitative Data 


Spatial Tracking |, Worker Temporal Use of Space __, Gathering information about the 


(Heatmap) amount of time spent in a place 


Fig. 1: VR safety training experience effectiveness evaluation approach — Data schema 


3.1. Post-experience evaluation questionnaire 


The approach adopted for the creation of the evaluation questionnaire led to the distinction of two main sections: 
the first section includes the trainee's personal data, while the second section corresponds to the actual evaluation 
part. The latter includes both a series of evaluation questions, for each of which the trainee can give a score on an 
increasing scale from one to five (where five represents maximum satisfaction), and an open "suggestions" section 
to collect trainees' personal impressions and ideas. Table 1 shows the sections and subsections with an indication 
of the purpose for which it was deemed necessary to include them, a brief description, and an explanatory example. 


3.2. Trainee’s spatial tracking and use of space visualization 


From the immersive VR experience, data are collected not only directly, by filling out questionnaires, but also 
indirectly with the acquisition of the worker's position in the virtual environment. The latter procedure involves 
capturing the position of the worker in the VR environment during the entire course of the simulation in order to 
analyse the actual use of the workspace and is recorded with a data acquisition rate of 1 Hz. The 3D point sequence 
resulting from this process is then converted for the generation of heatmaps within which the position of the worker 
is represented with a gradient of colour ranging from green to purple. 


As with the formulation of the questionnaire, mentioned in the previous paragraphs, an approach was defined to 
analyse worker movement during the immersive VR experience in order to visualize position tracking. This 
methodology involves taking into account not only the spatial coordinates, but also the time duration required for 
the performance of the experience. 
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Table 1: Questionnaire contents 


Data type 


Description 


Example 


ID & demographics 


The worker's data, which includes an identification code/number (ID) for 
classification of the completed questionnaire and an indication of the 
worker's age, role/occupation and company, with the purpose of clustering 
the results obtained. 


[Text] 


ID (Number) 


Age; Company; Role 


Immersivity 


Questions related to the user's ability to get immersed in the virtual 
experience, useful for assessing how engaged the trainee actually was in 
the virtual training scenario. It consists of an assessment of the trainee's 
satisfaction in terms of both ease of use and comfort (also related to the 
possible onset of symptoms of "motion sickness"). 


[Single rating — Likert scale] 


How much did you like the 


experience? 


[min 1; max 5] 


Contents’ 
effectiveness 


Questions regarding the virtual representation of the site and security 
content, aimed at evaluating the effectiveness of the experience against the 
objectives of the training session. 


[Single rating — Likert scale] 


How accurately does the virtual 
construction site reproduce the real 
one? 


[min 1; max 5] 


Training 
experience’s 
effectiveness 


Questions pertaining to the overall effectiveness of the experience, aimed 
at investigating the actual usefulness of the immersive VR training session 
and whether it turns out to be as comprehensive and exhaustive as a 
traditional training session. 


[Single rating — Likert scale] 


Do you think this experience is 
useful in understanding the hazards 
present on the construction site? 


[min 1; max 5] 


Future development 


Questions concerning the introduction of new automation or virtual content 
aimed at enhancing the immersive VR training experience through the 
inclusion of even virtual objects with enhanced interactivity. 


[Single rating — Likert scale] 


Would you like to be able to grasp 
and use objects from the 
construction site? 


[min 1; max 5] 


Suggestions Open question to collect personal impressions and ideas from the trainees. Suggestions 
[Text] 
Table 2: Heatmap schema contents 
Data type Description Example 
ID & demographics See Table 1 See Table 1 
Training experience The duration of the experience from start to finish, excluding the tutorial 523 sec 


duration 


that is run at the end to explain the operation to trainees, takes into account 
how long it takes the user to complete the experience. 


[time in seconds] 


Worker spatio- 
temporal track 


The graphical representation is given by a series of points that correspond 
to the coordinates of where the user was every | second and are used to 
create an image that shows the path followed by the worker distinguished 
with different colours. 


[Frequency in Hz] 
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Image with coloured representation 
of the worker’s followed path. 


[min. green, max. purple] 


4. IMPLEMENTATION 


The area in which this paper is located is part of a previous research project in which the authors' goal was to 
develop a prototype protocol for the design and delivery of safety training to construction workers, based on an 
innovative and interactive learner-centred approach. This protocol was designed and then tested for validation in 
a construction project in Italy that served as a case study for the development of VR training session content and 
implementation (Getuli et al., 2021). During and after the exploration of the site phases implemented, the worker 
undergoing the experience was invited to provide indications and opinions regarding the work environment in 
which he/she was working. This was done with the aim of both validating the different site layouts designed and 
to collect any objections and/or observations from the worker, thus enhancing their experience and giving it due 
importance within the development process of the virtual reality training experiences covered by this research 
work. 


In order to collect direct feedbacks and suggestions to better drive the decision of the development direction of the 
implementation of the proposed VR training protocol, an evaluation questionnaire (Fig.4) was administered from 
a staff member to every trainee involved after they finished their test training session. The authors drawn the 
questions in relation to the following development areas, weighting the number of the questions for each one 
according to their research objectives: 


e Immersivity: (question | to 4) Evaluation of the trainee’s satisfaction in terms of ease of use and comfort 
(also related to eventual “motion sickness” symptoms occurrence). 

ə Contents’ effectiveness: (question 5 to 10) Evaluation of the site’s and the safety contents’ virtual 
representation in respect of the purposes of the training session. 

e Training experience’s effectiveness: (question 11 to 14) Evaluation of the overall effectiveness of the 
immersive VR training experience. 

e Future development: (question 15 and 16) Evaluation of the introduction of audio and enhanced object’s 
interactivity as new features to be implemented in future developments. 


For each evaluation question the trainee can give a score on an upward scale from one to five, where five represents 
the highest satisfaction. Furthermore, an open “suggestions” section is added to collect personal impressions and 
ideas from the trainees. All the results were collected and processed in anonymous form. 


User Data 


ID: 

Age: 
Company: 
Role/Job: 


Questionnaire min - max 


> 


RAHA HAH HHH HAHAHAHA 
NANO nnranrnannanannanannn 


1) Did you enjoy the experience? 

2) Were you comfortable during the experience? 

3) Was it easy to use the viewer and controller? 

4) Did it bother you not being able to see your body in the virtual world? 

5) Are the signal arrows helpful in figuring out where to go? 

6) Does the virtual construction site sufficiently replicate the real one? 

Contents' 7) Do the work spaces indicated for the work seem adequate? 

Effectiveness 8) Are the safety and hazard spaces useful in signaling hazardous areas? 

9) In your experience, are the work procedures reproduced correct? 

10) Do you think this experience is useful in getting to know the worksite before entering it? 
11) Do you think this eperience is useful in understanding the hazards present at the worksite? 


Immersivity 


ere 12) Is this type of training useful for an inexperienced worker? 
: io . h 
Effectiveness 13) Is this type of training useful for an experienced worker? 


14) Would you prefer to see people and machines in motion ? 
Future < 15) Would you prefer to hear sounds and noises inside the worksite? 
Development 16) Would you like to be able to grasp and use worksite objects? 


RR ee ee ee ee 
NNNNNNNNNNNNNNDN WY 
WWWWWWwwwwww www w Ww 


Suggestions 


Fig. 2: Scheme of the VR training session evaluation questionnaire administered to the trainees 
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In addition to the data collected in active form through the questionnaires, an algorithm for recording the position 
of the worker in the virtual reproduction of the worksite was integrated into the VR training experience 
applications. In this way, during the training experience, with an acquisition frequency of 1 time per second (1 
Hz), the coordinates of the worker's position are recorded in relation to both the work spaces designed for the 
simulated construction activities, and in general in his movement within the construction site, so that his aptitude 
for recognising risk areas can be assessed a posteriori at both the site and activity scales. 


The interpretation of the movement traces acquired by the VR device during training in the form of sequences of 
points in the virtual space of the worksite was conducted by developing an additional analysis algorithm capable 
of reporting this information within the BIM model of the worksite and providing a graphic interpretation by means 
of heat-map visualisations. These visualisations allow the temporal dimension of the path followed by the worker 
in VR to be reported in a planimetric elaboration, distinguishing with different colours, in accordance with a pre- 
set gradient, the areas where the worker spent the longest time (violet, red) from those of short passage (green). 


The tracking of the worker's position recorded during the VR activity simulation is done by visualizing the worker's 
use of space based on a temporal heat map, which consists of a 2D representation of the 3D position points recorded 
by the worker. The time dimension of the position tracked by the worker is graphically represented through a 
colour gradient (green to red), so that a red-coloured area represents a previously recorded position (red indicates 
a position occupied longer during the VR simulation). The heat map is then automatically generated using a custom 
algorithm specially developed by the authors. 


Comparison of the generated heat map with the initial configuration of the workspace, in a BIM modeling 
environment, allows for early identification of possible planning errors; in fact, the results of the analysis of the 
obtained data are necessary for the next planning step, i.e., modification of the workspace configuration. 


: min sec 
[Duration | 00:00 | 000 


Site Scenario 
Plan 


Durata [s] 


101 


Space Usage 
Gradient 


Note 


Fig. 3: Heatmap visualization (plan view) of the trainee usage of the virtual training environment 
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5. CONCLUSIONS 


The present work, through implementation of an approach based on the integration of two main categories of data, 
with the overall objective of collecting a set of data from trainees experiencing immersive VR training to evaluate 
the effectiveness of the virtual experience, contributes to the development of a standardized method of evaluating 
the effectiveness of safety content transfer to workers that does not neglect the trainee's perceived effectiveness of 
the administered training and spatial understanding of the worksite scenario and activity in VR, as is the case with 
most of the methods proposed in the literature that are specific to individual case studies or applications and 
evaluate training based on an objective quantification of the trainee's ability to perform tasks in the VR 
environment. 


The developed questionnaire, dedicated to evaluating both the immersiveness of VR experiences and the 
effectiveness of the safety content presented, and the proposed study on the benefits of tracking and visualization 
of learners' use of space in the virtual environment, allowed us to conduct an evaluation of safety training 
experiences in VR for construction workers, based on the integration of both a qualitative survey of the experience 
and the participants’ spatial tracking data. Spatial tracking of trainees and their movement in space during the VR 
simulation of work activities proved useful for the evaluation of trainees’ spatial understanding of the VR safety 
training experience. 


Finally, the proposed approach was validated within a case study previously developed by the authors in which 
the authors' goal was to draft a prototype protocol for the design and delivery of safety training to construction 
workers, based on an innovative and interactive learner-centred approach. That protocol was designed and then 
tested for validation in a construction project in Italy that served as a case study for the development of BIM-based 
construction worker safety training in VR. During that work, a total of 6 VR training experiences were developed 
for workers, the contents of which consisted of the 3D models needed to reproduce the construction site scenario 
of the case study in the first 3 phases of construction: site set-up, installation of the external staircase and erection 
of the tower crane. At the same time, 4 training days were organized, during which the results of the proposed 
questionnaires were collected with reference to the different VR experiences carried out. 


During and at the end of each VR training session of the 6 different site phases implemented, the worker 
undergoing the experience was asked to fill out the questionnaire developed to provide input and opinions on the 
work environment they were in, for the evaluation of the session, in order to enhance the experience and collect 
useful data for the development of subsequent implementations. The results obtained from the above evaluation 
and the discussion of the related case study previously mentioned will be the subject of further publication. 
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